Split Federated Learning

This table illustrates the procedure of computation and communication for each epoch.

• client
• main server (stores part of model; do forward and backward propagation)
• fed server (solely for aggregating client-side models)

Clients	Server
Forwardprop @ local (in parallel)
Send activition to server
	Do rest of forwardprop (for each client)
	Do part of backprop (for each client)
	Send gradient to respective clients
	Update server model (e.g. agg. with FedAvg)
Do backprop, computes gradient
Send gradient back to server with DP
	Aggregate client gradients (e.g. with FedAvg)
	Send aggregated gradient to clients
Update param with received gradient

After several epoches, which is called a round, all client-side models are sent to fed-server for aggregation (e.g., with FedAvg).

SFLv1 is server-side aggregation based variant. It computes forward and backward propagation for each client in parallel and aggregates gradients with FedAvg to update server-side model *once.

In contrast, SFLv2 does not depend on aggregation. After receiving client-side activition, it sequentially processes the forward and backward propagation.

In machine learning, due to privacy concerns, algorithms are also developed to take cases that disabling label-sharing into account.

If label-sharing is allowed, then we can simply adopt SFLv1 or SFlv2. In cases that label-sharing is banned, we can make client process 2 model portions:

• part one: first few layers of the model \(W\)
• part two: last few layers of the model \(W\) and loss calculation.

This settings trade communication expense for better privacy concerns.